Skip to content

Add multi-model support infrastructure for SAFE framework#37

Open
Robbyswimmer wants to merge 17 commits into
testfrom
claude/multi-model-support-bVa0D
Open

Add multi-model support infrastructure for SAFE framework#37
Robbyswimmer wants to merge 17 commits into
testfrom
claude/multi-model-support-bVa0D

Conversation

@Robbyswimmer

Copy link
Copy Markdown
Owner

This commit introduces comprehensive support for using different backbone
models beyond LLaVA 1.5 13B, including:

Model Registry (safe/models/model_registry.py):

  • Centralized registry for all supported models
  • ModelSpec dataclass with architecture details
  • Support for Vision-Language, Text-Only, and API model types
  • Pre-registered models: LLaVA 1.5 (7B/13B), BLIP2, Llama 3.1/3.2,
    Mistral 7B, Qwen 2.5, Phi-3, Gemma 2, and API models (Gemini, GPT-4)

Base Model Abstraction (safe/models/base_model.py):

  • Abstract BaseModel class with unified interface
  • TextOnlyModel for Llama, Mistral, Qwen, Phi, Gemma, etc.
  • VisionLanguageModel for LLaVA, BLIP2
  • APIModel for Gemini Flash/Pro, GPT-4o, Claude
  • Factory function create_base_model() for easy instantiation
  • Support for Flash Attention 2 and quantization (8-bit, 4-bit)

Updated BaseVLModel (safe/models/base_vl.py):

  • Automatic model type detection from model name
  • Support for text-only models without vision encoder
  • New properties: has_vision, hidden_size, num_layers, num_attention_heads
  • get_decoder_layers() method for layer hook injection
  • Flash attention and quantization support for text-only models

Model Configurations (configs/model_configs.py):

  • Added 15+ new model configurations:
    • LLaVA 1.5 7B (smaller variant)
    • Llama 3.1 8B / 8B-Instruct
    • Llama 3.2 3B / 1B (small efficient models)
    • Mistral 7B v0.3
    • Qwen 2.5 7B
    • Phi-3 Mini 4K
    • Gemma 2 9B / 2B
    • Gemini Flash / Pro (API)
    • GPT-4o / GPT-4o Mini (API)
  • Each config includes optimized fusion layer indices and settings
  • Convenient aliases: llama-8b, llama-3b, llama-1b

This enables experimenting with modern LLMs like Llama 3.1 and smaller
efficient models for faster iteration and reduced memory requirements.

- Log evaluation entry point with GPU memory stats
- Log batch processing progress (every 10 batches)
- Log CIDEr/SPICE computation steps
- Free GPU cache before SPICE (Java subprocess)
- This will pinpoint where the OOM crash occurs
- Detect if captions are already in list format (audiocaps_val.json)
- Skip re-grouping if data is pre-grouped
- Add logging to show which file is loaded
- This should fix avg=1.0 references issue
@Robbyswimmer

Copy link
Copy Markdown
Owner Author

@codex review for issues

@chatgpt-codex-connector

Copy link
Copy Markdown

To use Codex here, create a Codex account and connect to github.

@Robbyswimmer

Copy link
Copy Markdown
Owner Author

@claude review for issues

Adds comprehensive support for different backbone models beyond LLaVA 1.5:

Model Registry (safe/models/model_registry.py):
- Centralized registry for all supported models with ModelSpec dataclass
- Support for Vision-Language, Text-Only, and API model types
- Pre-registered: LLaVA (7B/13B), BLIP2, Llama 3.1/3.2, Mistral, Qwen, Gemma

Base Model Abstraction (safe/models/base_model.py):
- Abstract BaseModel class with unified interface
- TextOnlyModel for Llama, Mistral, Qwen, Phi, Gemma
- VisionLanguageModel for LLaVA, BLIP2
- APIModel for Gemini, GPT-4, Claude
- Flash Attention 2 and quantization (8-bit/4-bit) support

Updated BaseVLModel (safe/models/base_vl.py):
- Auto model type detection via _detect_model_type()
- Text-only model support without vision encoder
- New properties: has_vision, hidden_size, num_layers, num_attention_heads
- get_decoder_layers() method for layer hook injection

New Model Configurations (configs/model_configs.py):
- llava-7b: LLaVA 1.5 7B (18GB VRAM)
- llama-3.1-8b: Llama 3.1 8B (20GB VRAM)
- llama-3.2-3b: Llama 3.2 3B (10GB VRAM)
- mistral-7b: Mistral 7B v0.3 (18GB VRAM)
- qwen-2.5-7b: Qwen 2.5 7B (18GB VRAM)
- gemma-2-9b: Gemma 2 9B (22GB VRAM)
- gemini-flash: Gemini 1.5 Flash (API)

This enables using modern text-only LLMs like Llama 3.1 8B for audio
fusion experiments without requiring vision capabilities.
@Robbyswimmer Robbyswimmer force-pushed the claude/multi-model-support-bVa0D branch from 51b582e to c6b6477 Compare January 11, 2026 05:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants